Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use eval_batch_size for AutoEncoder loss stats #861

Merged

Conversation

efajardo-nv
Copy link
Contributor

@efajardo-nv efajardo-nv commented Apr 10, 2023

Description

  • Updated AutoEncoder to use eval_batch_size instead of batch_size (training batch size) for calculating loss stats in get_anomaly_score_losses. Using eval_batch_size is more appropriate since it will not impact the final trained model and affect model convergence. The default eval_batch_size (1024) is larger than default batch_size (256) and results in a speedup in training postproc ([BUG]: Long production DFP training times #816)
  • Fixed a couple debug statements that were throwing string formatting errors.

Checklist

[x] I am familiar with the Contributing Guidelines.
[x] New or existing tests cover these changes.
[x] The documentation is up to date with these changes.

@efajardo-nv efajardo-nv added non-breaking Non-breaking change improvement Improvement to existing functionality labels Apr 10, 2023
@efajardo-nv efajardo-nv requested a review from hsin-c April 10, 2023 20:17
Copy link
Contributor

@hsin-c hsin-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good to me.

@efajardo-nv efajardo-nv marked this pull request as ready for review April 10, 2023 23:01
@efajardo-nv efajardo-nv requested a review from a team as a code owner April 10, 2023 23:01
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Does this close #816? If so, this PR should be linked to that issue.

morpheus/models/dfencoder/autoencoder.py Outdated Show resolved Hide resolved
morpheus/models/dfencoder/autoencoder.py Outdated Show resolved Hide resolved
@mdemoret-nv
Copy link
Contributor

/merge

@rapids-bot rapids-bot bot merged commit 59b7c06 into nv-morpheus:branch-23.07 Apr 17, 2023
@efajardo-nv efajardo-nv deleted the use-eval-batch-for-loss-stats branch July 29, 2024 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants